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ABSTRACT 

Tasks 5 and 6 describe the disseminat.t.on activities 
and a rating of the National Assessment for Educational Progress 
social studies exercises by members of the National Council' for the 
Social studies (NCSS) . The dissemination activities, described in a 
one page report, include a special issue of the NCSS journal ^social 
Education, »• Hay 1974, and a special brochure. The rest of the 
document consists of the Task 6 report. Nine NCSS members rated the 
appropriateness of the social studies exercises for the 1970»s. The 
panel rated the exercises according to the NCSS Curriculum Guidelines 
of knowledge, abilities, valuing, and social participation; the 
general desirability of the exercises based on the NCSS guidelines; 
and the realistic national level of performance that would satisfy 
the panel's professional judgement. The results for guidelines 
indicate a small percentage of exercises dealing with social 
participation, about one-fifth dealing with valuing, and a majority 
of 46 percent dealing with knowledge. In the general desirability 
Category less than 10 percent of the exercises have a rating of 
little or some value, while two-thirds of the exercises fall in the 
area of general desirability. In the category of estimation on the 
level of performance nationally, the panel thought many of the 
exercises were too difficult for many students. (DE) 
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FINAL REPORT ON TASK V-DISSEMINATION 

-4" The National Council for the Social Studies has carried out a number of 

dissemination activities. 

O' . 

y-H ^^^^ Project members have prepared two publications. 1) A special issue 

O of the NCSS journal, Social Education, May, 1974, went to all subscribers, who 
^ include not only NCSS members, but school libraries and curriculum centers, and 
collc?^^ and university libraries. 2) A Bulletin will be published by NAEP 
through the Government Printing Office. The two were developed with somewhat 
different emphases. The first focused on what could be learned from NAEP of 
direct relation to issues live in schools. The second emphasized reporting to 
the rrofession at large on matters basic to the National Assessment in Citizen- 
ship and Social Studios. 

The NCSS Project has also reported on the Citizenship and Social Studies 
* Assessments at NCSS annual meetings: in 1973 at three general sessions and one 
^^ch before the Social Studies Supervisors Association, the State Social Studies 
Specialists, and the College and University Faculty Assembly, all sub-groups of 
NCSS; and in 1974 at a general session. Reporting and discussion sessions were 
also set up at the 1974 annual meetings of AERA and ASCD. 

NCSS is also arranging to have Final Reports available through the NCSS cen- 
tral office and ERIC. 

Shifts in funding and a consequent train of events in NAEP had to delay the 
availability of Social Studies Assessment data. Much of the analysis of the 
findings could not be made during the formal lifetime of the Project and could 
not be disseminated. Hopefully more can be done in the future. 

Since NCSS interest in National Assessment will continue, some further 
dissemination activities will also continue. 

NCSS has welcomed its opportunity to work with the National Assessment of 
Educational Progress. That the relationship has been a productive one is a 
^ mark of the professionalism of NAEP and NCSS. 

Jean Fair, Chairman 
B£ST copy AV^IU|f]{ £ Steering Committee, NCSS Review Project 
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Social ijtuciies iOducatorr. Rate thn i.'Aci- 



Purposes 

How do social studies educators rate the National 
Assessment of rLducational Irorress (liAEf) social studies 
test exercises? To help answer this and other questions, 
N'ASr contracted with the Ilatlonal Council for the Social 
Studies (r:Co3). Representing^ NCSS was a steering committee 
with .jean Fair as chairperson. Other members of the 
committee were June Oiliiard, uana Kurfman, James 
Shaver, and Ronald Smith. 

In the spring of 1973 » this steerin*^, comraitlee 
of MCSS selected a panel of nine social studies educa- 
tors. In addition, the author was designated as 
chairperson with major responsibilities in designing 
rating sheets and processing the data. 

The main purpose of the panel, a group external 
to NASP, was to rate each IIAEP social studies exercise 
on the four age levels. They did this task by choosing 
the one best answer to each of the following questions 

1. Tnto which of the main components of the iiCSS 



Guidelines do you think hhe exercise best fits? 



("nowledr.e 



Abil i ties 



Valuinp" 




Social Participation 



2. Using the NC3S Guidelines as criteria, estimate 
the general desirability of the exercise* 

[. [ 'Little 
[ I Some 
I [.Moderate 
i iHi^h 

Very Great 

3. Realistically, what level of performance nationally 
for the age group being considered would satisfy 
you for this exercise. 

Perc en tage Expec t e d C orr ec t 




<^20 20-i^O 41-60 61-80 ^0 

In exercises in which there was more than one subpart, 
the panel rated each subpart separately. 

Rationale 

One concern of NAEP was the appropriateness of the 
NAEP test exercises for the 1970 's. NAEP had started 
its initial formulation of social studies objectives 
in 1965* To guide their effort in the formulation of 
social studies objectives, NAEP involved social scientistsi 
school peopls), and the community. The end result of 
NAEP efforts was the following five major objectives 
for the social studies t 
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1. Have curiosity about human affairs i 

2. Use analytic-scientific procedures effectively; 

3. Are sensitive to creative-intuitive methods of 
explaining the human condition} 

Have knowledge relevant to the major issues 
and concerns '*tf social scientistst 
5. Have a reasoned commitment to the values that 

sustain a free society (1). 
It appears that the MAEP social studies objectives 
defined the social studies as mainly a miniature version 
"of the social sciences. Using these five social Studies 
objectives, a contracting agency then designed test 
exercises representing samples of behavior from the 
universe of the stated objectives. 

In 1971, however, the National Council for the 
Social "Studies published its position paper entitled 
"Social Studies Curriculum Guidelines." (2) The 
committee which wrote the NGSS Curri culum Guidelines 
offered a somewhat different perspective of the social 
studies. The NCSS Curriculum Guidelines defined the 
social studies as essentially citizenship education 
with social problems emphasized as the major concern 
of the social studies curriculum. The NCSS position 
statement also advocated the importance of social 
participation in trying to resolve the problems con- 
fronting society. The committee which wrote the NCSS 



position paper identified the following four components 
as essential to the social studies curriculumt (1) 
Knowledge} (2) abilities} (3) valuing} and (^) social 
participation* 

The NCSS Curriculum Guidelines was viewed as a 
stimulus and guide for schools to evaluate their existing 
social studies curriculum and to work for improvement. 
The committee members largely responsible for the 
NGSS Curriculum Guidelines were higher education faculty 
members, in their thirties, who had recently received 
their social studies doctorates. Compared to the 
NAEP objectives, the NGSS Curriculum Guidelines may 
reflect more of the spirit of the times and be more 
at the dutting edge of where social studies is in the 
1970 's. Considerable overlap, however, does exist 
between the two different sets of objectives of the 
NCSS and the NAEP. 

To point out more precisely the relationship of the 
NAEP exercises to the NCSS Curriculum Guidelines , the 
first tasK of the panel members was to classify each 
exercise into one of the four main components stated 
in the NCSS Curriculum Guidelines . The panel's work on 
this task would givr some indication of the balance 
of the various NAEP test exercises as compared to the 
NCSS framework. Schools and communities, for example, 



would hardly be satisfied if almost all of the test 
exercises fell into j^st one UCSS category. 

Secondly, again using the liCSS Curriculum Guidelines 
as criteria, the panel members were to "estimate the 
general desirability of the exercise." Why was this done? 
It would be possible for the NAEP test exercises to have 
a good balance of the four basic liCSS categories (knov/ledge, 
abilities, valuing, and social participation) and still be 
considered of little value or not in keeping with what 
the NCSS Curriculum Guidelines recommended. Ey asking 
the panel to rate the general desirability of the exercise 
into one of five categories (little, some, moderate, high, 
very great), it was hoped that social studies professionals 
could give their estimations of the general desirability 
of each NAEP test exercise. 

Thirdly, the panel considered the question of 
"Realistically, what level of performance nationally 
for the age group being considered would satisfy you 
for this exercise?" Their professional judgment then 
could be used as a comparison with the actual levels 
of performance of students and adults. Unfortunately, 
due to delays in securing data from MAEP on actual 
student performance on the social studies test items, 
C. Benjamin Cox, the investigator assigned to this 
task, could not make the comparisons of the panel's 
ratings, and the student performance. 



Procedures 

The nine social studies professionals and the 
chairperson met in the office of UAEP. April IZ'1^» 
1973, Denver, Colorado. Their names and their 
professional status are listed below j 

Higher Education Faculty "embers 

1. Y. Arturo Cabrera, Bi-Lingual Education, 
University of Colorado 

2. Patricia Glasheen, Elementary Education 
Department, Rhode Island State College 

3. Jan Tucker, Secondary Education Department, 
Florida International University 

Public School Personnel 

1. Charles Beaty, Social Studies Consultant, 
Shawnee Mission (Kansas) Public Schools 

2. Lynda Falkenstein, Social studies Consultant, 
f.lultonmah County, Portland, Oregon 

3. Florence Jackson, Curriculum Supervisor, Bureau 
of Social Studies, New York City Public Schools 

Roosevelt Ratliff , Curriculum Supervisor, iv.ulti- 
Ethnic Studies, Tulsa Oklahoma Public Schools 

5. Zada Koblas, Teacher, ?v!ine^apolis Public Schools 
Doctoral Student 

1. Anne Powers, full-time doctorate graduate student, 
University of North Carolina 

What were the characteristics of the panel? 
Besides being a very able group with a varied geographic 
background, they were a relatively young groups— seven 
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under ^1-5 and three under 30 years of a^,e. The male- 
female ratio was about equal, in addition, one third 
were members of minority groups. The representation 
of minorities on the panel was especially important 
in light of some of the criticism that IIAEP test 
exercises are not appropriate to minority groups. 
Professionally, the panel was heavily represented 
by social studies consultants and higher education 
personnel, but underrepresented classroom teachers. 

A training session was first held using the NC3S 
Curriculum Guidelines and the rating forms. To insure 
tiat the panel understood the key ideas contained in 
the NC3S Curriculum Guidelines , an outline worksheet 
with the important phrases of the four NCS3 categories 
was distributed. For example. Social Participation 
was quoted on the worksheet as follows t 
Social Participation 

Extensive involvement by students ...in the 
activities of their community 

...may take the form of political campaigning, 
community service or improvement, or even 
responsible demonstrations 

*• .commitment in the social arena 

Released MASP citizenship test exercises were then 

used as sample exercises. Each member of the panel 

rated the sample question privately and then a discussion 

followed on the group's rating. An example of a sample 
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citizenship exercise is as followsj 

A. Have you ever spoken out in a public meeting to 
defend someone or some idea? (Yes, No) 

However, in working some sample exercises, individual 
members of the panel disagreed at times with the majority 
opinion. Jan Tucker, who had served as one of the four 
members of the MCSS position paper on Curriculum Guidelines 
and also was a member of the panel reviewing the NAEP 
social studies exercises, assisted in interpreting the 
NCSS Curriculum Guidelines * His comments were extremely 
helpful , but did not completely resolve all questions. 

The differences inthe panel members* ratings may 
be due to the ambiguity in meaning found in the UCSS 
Curriculum Guidelines , different interpretations of 
meaning in the test exercises, and individual differences 
among the raters. For example, in the sample exercise 
just given, panel members may have different interpreta- 
tions of what a public meeting is. Does a public meeting 
include student council meetings or do public meetings 
usually only include adults? The differing interpreta- 
tions of the term may then influence the panel's responses, 
especially with regard to what level of performance would 
satisfy them at the different age levels. 

After the training session, the raters proceeded 
at their own pace. The test exercises were only identified 



by a code number and the panel did not know what 
objective the test exercise was originally designed 
to represent. 

Results 

The panel rated each test exercise for each of the 
three tasks. Each panel member literally made over 
1,000 ratings since many test exercises had sub-parts» 
The data of the panel were processed. The computer 
print-out, totaling 2,992 pages, consisted of a histogram 
as well as the mean and standard deviation for each of 
the three tasks on every single test exercise. 

In the classification of the test exercises using 
the four main components of the UCS3 Curriculum Guidelines , 
the panel placed the MASP social studies exercises in 
the following categories (Table 1). 

y/hat interpretation can be made of the panel's 
classification usin?; the four categories of the 11033 
Curriculum Guidelines ? Striking is the very small 

/ 

percentage of test exercises that v/ere placed into the 
"Social Participation" category. This may be partly 
explained by the fact that liAEP is also responsible for 
assessing the area of citizenship. If the panel had also 
rated the citizenship exercises, the category "Social 
Participation" may well have increased* An examination 
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Table 1. Summary of l.AEP iJocial Studies 
Exercises Classified by i:G33 Guidelines 



N 0S3 Goniponent ?ercentar:e number of 



Knowledge 
Abilities 



Observations 

ii&.o 1,555 
31 t 1,041 



Valuing 19/'^ 646 

Social Participation 2';^ 53 

2;^ 2i 



No Response 
Total 



100/. 3.366 
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of the released citizenship exercises would appear to 
support this claim, liowever, the panel's ratings ^ 
probably do give some indication of liAlSF's view of 
the social studies. "Social I articipation" is not 
stressed in the l.AEP social studies test exercises. 

According to the panel's ratings . •'Abilities" plus 
-valuing" equaled about one-half of the social studies 
exercises with the "Knowledge" category receiving the. 
other half, ^^any may be very pleased that the ME? 
social studies exercises, according to the panel, do 
devote about one-fifth of their total to "Valuing." 
An important question is if the knowledge category 
is too high. In practice, the testing programs of 
most schools probably devote a much higher percentage 
of test exercises to the "Knowledge" category. Many 
teachers and the public may be very satisfied with the 
amount of attention devoted to the "Knowledge" category. 

Using ^h- t""^-^ r.nrriciilum Guidelines as criteria 
in estimating general desirability of the exercises, 
the panel rated the exercises in the following 
categories (Table ?.). 
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Table 2, Summary of NAI'lP Social Studies 
Exercis'es Classified by General Desirability 



De?.ree of Mumfcer of 

Desirability Percentaf^e Observations 

Very Great 26.?^. 879 

High ^0% It 356 

":oderat-2 21fa 722 



Some 



No Response 



Little 97 



L^'/. 118 



Total lOQfo 3»366 
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According to the panel, less than 10;'. of the social 
studies exercises have "Little" or "3omo" value according 
to the NC33 Curriculum Guidelines , If one uses the first 
two categories ("High" and "Very Great") as the criteria 
for desirability, the panel jud^!;ed about two-thirds of 
the HAEP exercises as fallin^;^ in the area of general 
desirability. If one defines desirability as including 
the first three categories ("High," "Very Great," and 
""'oderate"), then approximately 8?^ of the test 
exercises, using the panel's interpretation of the NCSS 
Curriculum Guidelines , are desirable. 

On the third task, an estimation on the level of 
performance nationally that the panel member would 
be satisfied with, the panel classified the social 
studies exercises as shown in Table 3» It would appear 
that the panel thought that many of the social studies 
exercises would be considered difficult for many students. 
In addition, the relationship of how the different 
age groups performed on the same or different test 
exercises merits important consideration, but was not 
part of the author's task. 



Table 3. Summary of NAEP Exercises 
Classified by Level of Performance 
Nationally that Would Satisfy Panel 
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Satisfaction Level of 
Panel with Regard to 
Percentaf?;e of Students 
Obtaining Correct 
Response 

Less than 20% 

Between 20 to ^o;^ 

Between ^1 to 60'/o 

Between 61 to 60;f 

Greater than 80:^ 

Total 



Percentage number of 

Observations 



2fo 

Wo 



lOOfo 



79 

186 

673 
1,858 

2.611 

5,407 
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Summary 

It should be emphasized strongly that the data 
presented in Tables 1, 2, and 3 are summary data, and 
therefore hide and gloss over many fine points in the 
individual test exercises. For example, on a single 
test exercise, the panel may have judged the item to 
be of little value in terms of desirability, but expected 
a large percentage of students to be able to answer the 
item correctly. On the other hand, the panel may have 
judged the exercise to have high value in terms of 
desirability, but would be satisfied if only a small 
percentage of students would be able to get the correct 
reponse. To make the best interpretation, it would be 
necessary to examine each exercise and the panel's 
rating of it. Again, individual examination of each 
test exercise fits in with NAEP's philosophy of looking 
at each test exercise rather than concentrating on 
summary data. 

The reliability of the panel was calculated for 
each test item. Sometimes the panel was unanimous in 
their classification of an exercise while at other times, 
they spread their responses across every available 
category. Even with a training session, it should be 
remembered that the nine panel members are human 
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bein^^s with different frames of reference, llut the 
fact that the panel differed on some test exercises 
should be an extremely fruitful area for further 
investigation. For example, it would be of interest 
to see how the minority members of the panel rated 
the exercises as compared to the rest of the panel* 
Unfortunately, sufficient computer funds were not 
available for the author to perform these operations. 

With these limitations kept in mind, it appears 
from the panel's ratings that the NAEP test exercises 
ai-e closely related to the philosophy of the NCSS 
Curriculum Guidelines (with the exception of the 
category. Social Participation), possess some obvious 
degree of desirability according to the NCSS Curriculum 
Guidelines , and professionals would be satisfied if 
more than 61% of the students were able to get the 
correct response on most iIAEP social studies exercises. 
However, it would seem very important that IIAEP look 
very carefully at exercises that the panel judged to 
be of little value in terms of desirabili-Jy. The 
investigator regrets that there was no available budget 
to pursue this area of investigation. 
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